Skip to content

Custom repos in benchpark#1190

Merged
scheibelp merged 47 commits intodevelopfrom
custom-repos
Jan 28, 2026
Merged

Custom repos in benchpark#1190
scheibelp merged 47 commits intodevelopfrom
custom-repos

Conversation

@scheibelp
Copy link
Collaborator

@scheibelp scheibelp commented Jan 16, 2026

Define a repos.yaml like:

repos:
  experiments:
  - ../experiments
  - ../experiments/test
  # and systems, applications, and packages, see config/repos.yaml added in this PR for full example

in <benchpark_root>/config to customize where experiment definitions are pulled from.

Benchpark can pull config from one location, with the following priority (highest first):

  • benchpark -C <dir>...
  • If CWD where you call benchpark has benchpark-config directory
  • If <benchpark_root>/config is a directory (this must exist if the first two don't)

More generally:

  • Add a rudimentary YAML-based config system to benchpark
  • Define a Repos section
    • Benchpark now uses this repos config to instantiate experiments and systems repos for benchpark itself, and packages/applications repos for Spack/Ramble
  • Accommodates other possible sections
    • These sections support property-like access in python (but that assume the property exists, which they all should in the case of repos config)

TODOs

  • several files are called repos.yaml or repo.yaml w/ different purposes, maybe I should call this new one something different
  • schema validation
  • Consolidate with benchpark bootstrapping config (currently gets consumed via a different mechanism, lives in a file in configure.yaml)

@github-actions github-actions bot added feature New feature or request experiment New or modified experiment labels Jan 16, 2026
@scheibelp scheibelp changed the title [WIP] Custom repos in benchpark Custom repos in benchpark Jan 20, 2026
Copy link
Collaborator

@michaelmckinsey1 michaelmckinsey1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From our discussion

Regarding the benchpark configure comand - take benchpark.yaml rename it to bootstrap.yaml benchpark configure will write to the config directory.

@github-actions github-actions bot added the ci CI, unit tests, GitHub actions label Jan 21, 2026
@github-actions github-actions bot added the docs Improvements or additions to documentation label Jan 22, 2026
Copy link
Collaborator

@michaelmckinsey1 michaelmckinsey1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(1) I think we need docs to instruct users to add the path to their experiments/systems/repo to config/repos.yaml. And that if they wish for their addition to be used over the existing version, it should go first in the config.

(2) Can you merge develop and run a gitlab pipeline?

Copy link
Collaborator

@michaelmckinsey1 michaelmckinsey1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some docs edits

@michaelmckinsey1 michaelmckinsey1 self-requested a review January 27, 2026 23:33
Copy link
Collaborator

@michaelmckinsey1 michaelmckinsey1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good now

@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 83.70787% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.45%. Comparing base (e2fef44) to head (5f2e26e).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
lib/main.py 60.86% 9 Missing ⚠️
lib/benchpark/config.py 88.70% 7 Missing ⚠️
lib/benchpark/base_paths.py 85.18% 4 Missing ⚠️
lib/benchpark/cmd/mirror.py 25.00% 3 Missing ⚠️
lib/benchpark/cmd/system.py 33.33% 2 Missing ⚠️
lib/benchpark/cmd/unit_test.py 33.33% 2 Missing ⚠️
lib/benchpark/accounting.py 80.00% 1 Missing ⚠️
lib/benchpark/cmd/info.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1190      +/-   ##
===========================================
+ Coverage    63.70%   64.45%   +0.75%     
===========================================
  Files           46       48       +2     
  Lines         3507     3618     +111     
  Branches       265      273       +8     
===========================================
+ Hits          2234     2332      +98     
- Misses        1264     1277      +13     
  Partials         9        9              
Files with missing lines Coverage Δ
experiments/test/saxpy/experiment.py 50.00% <ø> (ø)
lib/benchpark/cmd/setup.py 88.48% <100.00%> (+1.08%) ⬆️
lib/benchpark/cmd/show_build.py 21.42% <100.00%> (ø)
lib/benchpark/directives.py 71.07% <ø> (-0.24%) ⬇️
lib/benchpark/paths.py 100.00% <100.00%> (+14.28%) ⬆️
lib/benchpark/repo.py 73.84% <100.00%> (+1.53%) ⬆️
lib/benchpark/runtime.py 92.10% <100.00%> (ø)
lib/benchpark/test/caliper.py 97.43% <100.00%> (ø)
lib/benchpark/test/commands.py 100.00% <100.00%> (ø)
lib/benchpark/test/paths.py 100.00% <100.00%> (ø)
... and 8 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@scheibelp scheibelp added this pull request to the merge queue Jan 28, 2026
Merged via the queue into develop with commit 524fc9a Jan 28, 2026
22 checks passed
@scheibelp scheibelp deleted the custom-repos branch January 28, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI, unit tests, GitHub actions docs Improvements or additions to documentation experiment New or modified experiment feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants